A Method for Improving Automatic Word Categorization

A METHOD FOR IMPROVING AUTOMATIC WORD CATEGORIZATION Korkmaz, Emin Erkan MS., Department of Computer Engineering Supervisor: Ass. Prof. Dr. G okt urk  U coluk September 1997, 57 pages In this thesis study a new approach to automatic word categorization which improves both the e ciency of the algorithm and the quality of the formed clusters is presented. The unigram and the bigram statistics of a corpus of about two million words are used with an e cient distance function to measure the similarities of words, and a greedy algorithm to put the words into clusters. The notions of fuzzy clustering like cluster prototypes, degree of membership are used to form up the clusters. Di erent distance metrics are analyzed using the algorithm. Empirical comparisons are made in order to support the discussions proposed for the type of distance metric that would be most suitable for measuring the similarity between linguistic elements. The algorithm is of unsupervised type and the number of clusters are determined at run-time.